Effects of Position Bias on Click-Based Recommender Evaluation
نویسندگان
چکیده
Measuring the quality of recommendations produced by a recommender system (RS) is challenging. Labels used for evaluation are typically obtained from users of a RS, by asking for explicit feedback, or inferring labels from implicit feedback. Both approaches can introduce significant biases in the evaluation process. We investigate biases that may affect labels inferred from implicit feedback. Implicit feedback is easy to collect but can be prone to biases, such as position bias. We examine this bias using click models, and show how bias following these models would affect the outcomes of RS evaluation. We find that evaluation based on implicit and explicit feedback can agree well, but only when the evaluation metrics are designed to take user behavior and preferences into account, stressing the importance of understanding user behavior in deployed RSs.
منابع مشابه
User Behavior and Bias in Click-Based Recommender Evaluation
Measuring the quality of recommendations produced by a recommender system (RS) is challenging. Labels used for the evaluation are typically obtained from users of a RS; such explicit labels reflect true user preferences but may introduce significant biases in the evaluation process. In this paper, we investigate biases that may affect labels inferred from implicit feedback, such as clicks or ot...
متن کاملA Study of Position Bias in Digital Library Recommender Systems
“Position bias” describes the tendency of users to interact with items on top of a list with higher probability than with items at a lower position in the list, regardless of the items’ actual relevance. In the domain of recommender systems, particularly recommender systems in digital libraries, position bias has received little attention. We conduct a study in a real-world recommender system t...
متن کاملWhat Recommenders Recommend - An Analysis of Accuracy, Popularity, and Sales Diversity Effects
In academic studies, the evaluation of recommender system (RS) algorithms is often limited to offline experimental designs based on historical data sets and metrics from the fields of Machine Learning or Information Retrieval. In real-world settings, however, other businessoriented metrics such as click-through-rates, customer retention or effects on the sales spectrum might be the true evaluat...
متن کاملRevisiting the Examination Hypothesis with Query Specific Position Bias
Click through rates (CTR) offer useful user feedback that can be used to infer the relevance of search results for queries. However it is not very meaningful to look at the raw click through rate of a search result because the likelihood of a result being clicked depends not only on its relevance but also the position in which it is displayed. One model of the browsing behavior, the Examination...
متن کاملCharacterize Search Intent Diversity into Click Models
Modeling a user’s click-through behavior in click logs is a challenging task due to the well-known position bias problem. Recent advances in click models have adopted the examination hypothesis which distinguishes the document relevance from the position bias. In this paper, we revisit the examination hypothesis and observe that user clicks cannot be completely explained by the relevance and th...
متن کامل